Skip to content

Add retention sweep metrics for cycle duration, rows deleted, and consecutive failures - #5831

Merged
johnsimons merged 7 commits into
masterfrom
john/retention_metrics
Aug 31, 2026
Merged

Add retention sweep metrics for cycle duration, rows deleted, and consecutive failures#5831
johnsimons merged 7 commits into
masterfrom
john/retention_metrics

Conversation

@johnsimons

Copy link
Copy Markdown
Member

No description provided.

@warwickschroeder warwickschroeder left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does docs/telemetry.md need updating?

Comment thread src/ServiceControl.Persistence.EFCore/Infrastructure/RetentionSweeper.cs Outdated
Comment thread src/ServiceControl.Persistence.EFCore/Infrastructure/RetentionMetrics.cs Outdated
Comment thread src/ServiceControl.Persistence.EFCore/Infrastructure/RetentionSweeper.cs Outdated
…secutive failures

Persisters publishing onto a host's already-registered meter cannot reference the host assembly, so meter names are extracted into a shared ServiceControlMeters constants class.

Cycles interrupted by cancellation are excluded from measurements, as a clean shutdown is not a failure worth alerting on.
/// One pass of the retention sweep. A pass interrupted by shutdown is not a measurement of
/// anything, so a cancelled cycle records neither a duration nor a failure.
/// </summary>
public sealed class RetentionCycleMetrics : IDisposable

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should these classes be in separate files, and possibly in their own namespace/folder?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

this.cancellationToken = cancellationToken;
}

public void Complete() => completed = true;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know it's probably not material but shouldn't this also freeze the timer in case there is time between the Complete() and Dispose()?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good call, done. Complete() stops the stopwatch and Dispose() records the frozen value.

Covers the three metrics (cycle duration, rows deleted, consecutive failures), their tags, example PromQL queries, and guidance on reading the shapes — including what a climbing consecutive-failures gauge means for body store permissions and why passes are isolated from one another.
@johnsimons
johnsimons merged commit 1727928 into master Aug 31, 2026
102 of 104 checks passed
@johnsimons
johnsimons deleted the john/retention_metrics branch August 31, 2026 08:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants